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I.  Evaluating  Uncertainty  vs  Evaluating  Fusion 

Information  fusion  “combine[s]  information  from  multiple 
sources...  to  achieve  inferences  that  cannot  be  obtained  from  a 
single  sensor  or  source,  or  whose  quality  exceeds  that  of  an 
inference  drawn  from  any  single  source.”  [1,  p.  xiii\  By  the 
very  nature  of  the  fusion  problem,  an  ability  to  cope  with 
uncertainty  is  a  fundamental  requirement  for  a  fusion  system. 

Uncertainty  processing  is  built  into  standard  low-level 
fusion  algorithms.  For  example,  error  ellipses  are  a 
fundamental  ingredient  of  data  association  and  state  updating 
algorithms  for  multitarget  tracking  systems.  These  systems  take 
uncertainty  as  a  given;  fusion  serves  to  reduce  the  uncertainty 
by  combining  reports  in  a  way  that  accounts  properly  for  their 
individual  and  joint  uncertainty. 

At  Levels  2  and  above  in  the  JDL  hierarchy,  recognition  of 
importance  of  uncertainty  remains,  but  there  is  far  less 
commonality  in  approaches  both  for  reasoning  with  uncertainty 
and  for  evaluating  the  impact  of  these  approaches  on  the  fusion 
process.  There  is  a  bewildering  variety  of  information  sources 
and  types  of  outputs  for  high-level  fusion  systems. 
Characterizing  their  associated  uncertainties  and  assessing  their 
impact  is  a  daunting  challenge.  Furthermore,  especially  at 
higher  JDL  levels,  we  must  consider  not  just  uncertainty  related 
to  individual  information  sources  and  hypotheses,  but  also  how 
uncertainty  propagates  through  chains  of  indirect  evidential 
support.  In  addition,  it  is  often  important  to  represent  and 
reason  with  ancillary  evidence,  or  evidence  about  the  nature  and 
force  of  an  evidential  relationship  [2]. 


Evaluating  the  uncertainty  management  aspect  of  a  fusion 
system  is  only  one  aspect,  albeit  important,  of  evaluating  the 
fusion  system  as  a  whole  [3].  How  uncertainty  management 
contributes  to  performance  of  the  fusion  system  is  the  key  issue 
addressed  by  the  ISIF  working  group  on  evaluation  of 
techniques  for  uncertainty  representation  (ETURWG).  To  that 
end,  the  ETURWG  developed  an  ontology  designed  to  capture 
the  concepts  relevant  to  evaluating  uncertainty  of  information 
fusion  systems  [3].  The  Criteria  class  of  the  URREF  ontology 
captures  key  measures  relevant  to  how  a  fusion  system 
represents  and  reasons  with  uncertainty,  and  how  uncertainty 
contributes  to  performance  of  the  sytssem  as  a  whole. 

II.  Evaluation  for  Purpose 

In  examining  the  role  of  uncertainty,  it  is  essential  to  keep 
in  mind  the  purpose  for  which  fusion  is  being  performed. 
Generally,  this  is  to  provide  the  necessary  information  inputs  to 
support  some  kind  of  decsion.  Effective  information  fusion 
reduces  uncertainty,  but  uncertainty  reduction  comes  at  a  cost. 
How  much  and  what  kind  of  uncertainty  reduction  is  worth 
achieving  depends  on  how  the  results  will  be  used.  As  a  simple 
example,  suppose  we  are  interested  in  the  whereabouts  of  John, 
who  is  a  suspect  in  an  assassination  that  occurred  on  April  3.  If 
the  purpose  is  to  provide  an  alibi,  then  “John  was  somewhere  in 
France  on  April  3”  is  an  accurate  enough  localization  if  the 
assassination  occurred  in  New  York,  but  not  if  it  occurred  in 
Paris.  As  another  example,  typing  an  object  as  a  hostile  military 
vehicle  may  be  sufficient  identification  for  targeting  under 
some  rules  of  engagement;  while  a  more  precise  type 
identification  and/or  intent  assessment  may  be  necessary  under 
different  rules  of  engagement.  These  examples  illustrate  the 
principle  that  designing  and  evaluating  the  uncertainty 
management  component  of  fusion  systems  requires 
understanding  how  the  fused  results  will  be  used.  Different 
design  tradeoffs  are  appropriate  for  different  end  uses.  Distinct 
metrics  of  output  quality  and  different  thresholds  of  a  given 
metric  may  be  appropritate  for  different  purposes. 

A  basic  premise  of  fusion  is  that  combining  noisy  inputs 
from  multiple  sources,  especially  sources  with  uncorrelated 
errors,  can  give  higher  quality  results  than  the  individual 
inputs.  A  corollary  is  that  reducing  the  uncertainty  of  individual 
inputs  to  a  fusion  system  may  have  lower  payoff  than 
increasing  the  number  of  different  kinds  of  inputs  [4].  That  is, 
fusing  many  cheap  but  inaccurate  inputs  often  gives  higher 
quality  outputs  than  fusing  fewer  but  more  accurate  inputs. 


In  a  related  vein,  for  some  higher-level  fusion  problems, 
there  may  be  diminshing  returns  from  greater  accuracy  in 
inputs  from  low-level  fusion  systems.  For  example,  a 
simulation  study  by  Wright  et  al.  [5]  found  that  the  fidelity  of 
situation  estimates  was  relatively  insensitive  to  variations 
(within  a  reasonable  range)  in  some  kinds  of  low-level  fusion 
error.  In  particular,  platoons  could  be  reliably  identified  despite 
a  moderate  probability  of  missed  detections,  false  alarms  and 
incorrect  associations.  It  is  important  for  system  designers  to 
understand  how  errors  and  imprecision  in  low-level  fusion 
results  contribute  to  errors  and  imprecision  in  results  of  direct 
interest  to  decision  makers.  Such  understanding  enables  more 
informed  design  tradeoffs. 

III.  Measuring  the  Value  of  Information  Fusion 

The  concept  of  Value  of  Information  from  decision  theory 
provides  a  useful  conceptual  framework  for  evaluating  the  role 
played  by  uncertainty  management  in  fusion  systems.  In 
decision  theory,  we  model  a  decision  situation  as  consisting  of 
a  set  {d'.  dS)]  of  allowable  options,  a  set  {s:  s'GS}  of  possible 
world  states,  a  probability  distribution  P{s\d)  for  the  world 
state  given  each  allowable  option,  and  a  utility  function  u{s) 
representing  the  value  to  the  decision  maker  of  each  possible 
world  state.  We  may  decompose  the  world  state  s=(r,e,y)  as  a 
result  r  of  direct  interest,  observable  evidence  e,  and 
unobserved  but  aspects  y  of  interest  only  insofar  as  they  are 
relevant  to  the  result.  The  utility  u{s)  =  u{r)  depends  only  on  the 
result  of  direct  interest. 

The  decision  maker’s  optimal  choice  is  to  maximize 
expected  utilility.  With  no  evidence,  the  expected  utility  is: 

M*  =  max{£p(,j[M(r)]} 

where  the  expectation  is  taken  with  respect  to  the  distribution 
p{s)  =  p{r,e,y).  Now,  suppose  evidence  e  (which  in  general 
may  include  multiple  items  of  evidence  from  different  sources) 
is  processed  by  the  fiision  system,  resulting  in  an  updated 
probability  distribution  p{s\e)  that  incorporates  the  new 
information.  The  expected  utility: 

M**(e)  =  max{£^„,^)[M(r)]} 

is  a  function  of  the  evidence  e.  Prior  to  observing  e,  our 
expected  gain  in  utility  from  the  observation  is  the  expected 
value  of  sample  information  [6] : 

EVSI  =  £^(,)[M**(e)-«*] 

EVSI  is  always  non-negative,  and  is  positive  when  the 
information  has  the  potential  to  improve  the  decision.  Thus, 
EVSI  provides  “simple  and  elegant  measure’’  (c.f  [7])  to 
evaluate  the  contribution  of  information  fusion.  Conceptually, 
we  can  use  EVSI  to  compare  two  fusion  systems  according  to 
their  contribution  to  improving  decision-making. 

It  may  be  extraordinarily  difficult  or  even  infeasible,  both 
computationally  and  from  a  domain  modeling  perspective,  to 
quantify  EVSI  explicitly.  Nevertheless,  the  idea  of  EVSI  can 
help  to  inform  thinking  about  the  value  of  a  fusion  system. 
Furthermore,  there  have  been  many  advances  recently  in 


tractable  exact  and  approximate  computation  of  value  of 
information  [7].  Evaluating  fusion  systems  from  the 
perspective  of  EVSI  can  capture  aspects  of  the  mission  that 
cannot  be  captured  from  more  traditional  measures  of 
information  gain  (e.g.,  [8]).  Turning  to  a  comparison  of 
different  methods  of  fusing  information,  decision  theory 
provides  a  conceptual  framework  for  measuring  their  relative 
contributions.  Specifically,  we  can  compare  them  according  to 
the  expected  utility  each  system  achieves. 

IV.  Practicalities 

For  fusion  systems  of  realistic  complexity,  computation  of 
the  value  of  fusion  will  typically  be  infeasible.  However,  it  is 
useful  to  conceive  of  evaluation  as  an  attempt  to  measure  the 
contribution  of  an  uncertainty  management  approach  to  the 
overall  performance  of  the  fusion  system.  We  can  develop 
metrics  to  evaluate  different  aspects  of  performance  (as  laid  out 
in  [3])  and  consider  how  to  measure  these  aspects  in  practice. 
Even  when  there  is  no  agreement  among  stakeholders  on  how 
to  combine  the  factors  into  a  single  overall  utility  function, 
considering  the  factors  separately  can  give  insight  into  the 
strengths  and  weaknesses  of  different  approaches. 

If  the  objective  is  to  compare  different  uncertainty 
management  methods,  there  is  no  substitute  for  direct 
comparison  of  performance  on  common  data  sets.  Obtaining 
ground  fruth  for  high-level  fusion  problems  is  typically  quite 
challenging.  Comparison  on  simulated  data  sets  may  be  the 
only  option,  and  even  that  may  be  difficult.  Still,  if  the 
community  is  to  advance,  it  is  necessary  to  develop  a  set  of 
benchmark  problems  and  data  sets.  A  model  might  be  the  UC 
Irvine  machine  learning  repository  [9]. 
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